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Abstract: 

The Arts Achieve: Impacting Student Success in the Arts project involves a partnership between 
the New York City Department of Education (NYCDOE) and five of the city’s premier arts 
organizations. Arts Achieve provides intensive and targeted professional development to arts 
teachers over a three-year period. The goal of the project is to improve the quality of arts teachers’ 
instruction through in-service professional development on the use of balanced (formative 
and summative) assessment, leading to increases in students’ arts achievement. Starting in 
the 2011-2012 school year, arts teachers formed art discipline-based professional learning 
communities (PLCs) to work together, using a process of inquiry and action research that focuses 
on reviewing student data and examining impact on current instructional practice. Additionally, 
each arts teacher was paired with a facilitator from the arts organizations to support them 
over the course of the project. The specific professional development activities included: on-site 
consultancies, assessment retreats, inter-visitations, and an online community. Arts Achieve also 
provides participating arts teachers with resources to support this work, such as units of study 
and technology bundles. 

To measure the impact of the Arts Achieve project on arts teachers and students, Metis 
Associates designed a cluster randomized control trial study, whereby 77 schools were assigned 
to treatment or status-quo control conditions by arts discipline (dance, music, theater, visual arts) 
and school level (elementary, middle, high). In the planning year of the project, Benchmark Arts 
Assessments were developed in each arts discipline and school level to measure students’ arts 
achievement. Findings from Year 1 indicate that, while there were not statistically significant 
differences between the growth of treatment and control teachers, the students of treatment 
teachers demonstrated significantly greater growth in arts achievement from the students of 
control teachers. The results suggest that a more sensitive tool for detecting change in teachers is 
needed. Successes and challenges of project implementation are discussed, and potential areas 
for additional inquiry in the coming years of the grant also are recommended. 

Copyright Information: 

All rights reserved unless otherwise indicated. Contact the author or original publisher for any 
necessary permissions. eScholarship is not the copyright owner for deposited works. Learn more 

at http://www.escholarship.org/help copvriqht.html#reuse 


eScholarship 

University of California 


eScholarship provides open access, scholarly publishing 
services to the University of California and delivers a dynamic 
research platform to scholars worldwide. 




Mastrorilli et al.: Arts Achieve, Impacting Student Success in the Arts: Preliminary Findi... 


Arts Achieve, Impacting Student Success in the Arts: Preliminary Findings After One Year 

of Implementation 


Tara M. Mastrorilli, Susanne Harnett, Jing Zhu 


Abstract 

The Arts Achieve: Impacting Student Success in the Arts project involves a partnership 
between the New York City Department of Education (NYCDOE) and five of the city’s premier 
arts organizations. Arts Achieve provides intensive and targeted professional development to arts 
teachers over a three-year period. The goal of the project is to improve the quality of arts 
teachers’ instruction through in-service professional development on the use of balanced 
(fonnative and summative) assessment, leading to increases in students’ arts achievement. 
Starting in the 2011-2012 school year, arts teachers formed art discipline-based professional 
learning communities (PLCs) to work together, using a process of inquiry and action research 
that focuses on reviewing student data and examining impact on current instructional practice. 
Additionally, each arts teacher was paired with a facilitator from the arts organizations to support 
them over the course of the project. The specific professional development activities included: 
on-site consultancies, assessment retreats, inter-visitations, and an online community. Arts 
Achieve also provides participating arts teachers with resources to support this work, such as 
units of study and technology bundles. 

To measure the impact of the Arts Achieve project on arts teachers and students, Metis 
Associates designed a cluster randomized control trial study, whereby 77 schools were assigned 
to treatment or status-quo control conditions by arts discipline (dance, music, theater, visual arts) 
and school level (elementary, middle, high). In the planning year of the project, Benchmark Arts 
Assessments were developed in each arts discipline and school level to measure students’ arts 
achievement. Findings from Year 1 indicate that, while there were not statistically significant 
differences between the growth of treatment and control teachers, the students of treatment 
teachers demonstrated significantly greater growth in arts achievement from the students of 
control teachers. The results suggest that a more sensitive tool for detecting change in teachers is 
needed. Successes and challenges of project implementation are discussed, and potential areas 
for additional inquiry in the coming years of the grant also are recommended. 
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Introduction 

Teaching and learning in the arts, like any subject, requires the use of assessment to allow 
teachers to reflect on students’ progress and adjust their instruction to meet students’ needs. 
Assessment also provides students with feedback, which allows them to reflect on their own 
learning. More specifically, current research highlights the importance of balanced assessment 
(fonnative and summative) in instruction to support student learning and increase student 
achievement (Black & Wiliam, 1998; Hattie & Temberley, 2007; Kluger & DeNisi, 1996). In 
general, however, the arts lack access to high quality assessments to inform arts teachers about 
their students’ knowledge and skills in a particular art form. The Arts Achieve: Impacting Student 
Success in the Arts project was conceptualized to address this issue and, in doing so, improve 
teacher instruction and student learning in the arts. 

In spring 2010, the Arts Achieve project was funded by two United States Department of 
Education (USDOE) grants: the New York City Department of Education (NYCDOE) Office of 
Arts and Special Projects (OASP) received an Arts Education Model Development and 
Dissemination (AEMDD) grant and Studio in a School (STUDIO) received an Investing in 
Innovation (i3) grant to implement the program. The goal of Arts Achieve is to improve the 
quality of arts teachers’ instruction through intensive and targeted professional development on 
the use of balanced assessment, leading to increases in students’ arts achievement. To evaluate 
the efficacy of the Arts Achieve project, the project partners hired Metis Associates, a national 
evaluation and consulting firm, to conduct an experimental study, currently underway. At the end 
of the first project year, which was used as a planning year, 77 NYC public schools were 
assigned to treatment or status-quo control conditions using a stratified (art discipline and school 
level) random assignment process. In this article, we report data on the preliminary effects of the 
Arts Achieve project on arts teachers’ instructional practices, as well as on students’ arts 
achievement, after one full year of implementation. 

Theoretic Framework 

The logic model, which is displayed in Figure 1, illustrates the theoretical framework of 
the Arts Achieve project. The project is based on the theory that, when arts teachers participate in 
professional development, including being members of professional learning communities 
(PLCs), that focuses on action research and the use of balanced assessment strategies, the quality 
of their arts instruction will improve. In turn, the project hypothesizes that enhanced arts 
instruction will lead to improved student achievement in the arts. The paragraphs below provide 
an overview of research on these components, which have guided the work of Arts Achieve. 

Arts Achieve draws on current research around professional development for educators, 
which emphasizes the need for PLCs to develop the collective capacity of staff to work together 
in order to improve teacher practice and student learning. Through PLCs, teachers work together 
in teams, learning from each other in environments that are both school-based and job embedded 
(DuFour, Eaker, DuFour, 2005). The development of PLCs in the arts community is particularly 
needed, given that arts teachers can be the only staff members in their schools teaching in their 
content area, and many have reported that they do not feel integral to the faculty (Burnaford, 
2009). 

Several research studies provide robust support regarding the impact of PLCs on teacher 
practices, school culture, and student achievement. For example, Louis and Marks (1998) 
conducted a multi-site study in 24 schools on the impact of PLCs. Through classroom 
observations and interviews with teachers, they documented the presence of authentic pedagogy 
(i.e., pedagogy that emphasizes higher order thinking, the construction of meaning through 
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conversation, and the development of depth of knowledge that has value beyond the classroom) 
and examined the connection between the quality of classroom pedagogy and the existence of 
core characteristics of a PLC. This study found that the presence of a professional community in 
a school contributes to higher levels of social support for achievement and higher levels of 
authentic pedagogy. Bolam, McMahon, Stoll, Thomas, & Wallace (2005) examined survey data 
from 393 schools and interview-based case study data from 16 school sites. Both the survey and 
case study data revealed that when teachers work in PLCs, there are fundamental shifts in the 
ways that they approach their work, increasing their collaboration, reducing their feelings of 
isolation, and improving their overall morale. 

There is also strong evidence that when teachers work in PLCs, their students experience 
increased achievement. Eight recent studies (Berry, Johnson, & Montgomery, 2005; Bolam et al., 
2005; Hollins, McIntyre, DeBose, Hollins, & Towner, 2004; Louis & Marks, 1998; Phillips, 
2003; Strahan, 2003; Supovitz, 2002; Supovitz & Christman, 2003) all found positive 
relationships between teachers’ participation in PLCs and student academic achievement. In 
these studies, results of student achievement gains varied with the strength of the PLC in the 
school (Bolam et al., 2005; Louis & Marks, 1998) or with the specific focus of the efforts of 
teams or small communities of teachers (Supovitz, 2002; Supovitz & Christman, 2003). 

Underlying all of the work of the PLCs is an emphasis on action research and use of 
appropriately balanced assessment strategies to review and discuss student work and teacher 
practice. Action research involves teachers engaging in inquiry and reflection on their current 
practice and student work. In contrast to one-day professional development sessions, 
professional development that involves action research is more participant-driven and 
incorporates inquiry and reflection that occurs over a period of time. Zeichner (2003) notes that 
these components are in alignment with the standards and guiding principles for professional 
development as set by national organizations (e.g., the American Federation of Teachers, the 
National Councils of Teachers of English and Mathematics) and academic scholars (e.g., 
Darling-Hammond & McLaughlin, 1996). 

Studies have found that teachers who engage in action research report that they have 
higher self-esteem and confidence levels (Dadds, 1995; Loucks-Horsely, Hewson, Love, & 

Stiles, 1998), develop self-analysis skills that are applied to their teaching (Day, 1984), and 
become more aware of how they impact their students (Allen, Shockley, & Baumann, 1995). In a 
review of studies on four action research professional development programs, Zeichner (2003) 
reported that engaging in action research helps teachers create a more student-centered 
environment, in which the teachers focus on listening to and observing their students to influence 
instruction. He explained further that teachers begin to see the point of view of their students and 
allow them to have more input in the classroom. 

In order to inform teachers’ action research, balanced assessment is a critical tool to 
gather evidence on student progress. Current research emphasizes the need for teachers to use 
assessment as a tool to help gather student performance data and assess how they should target 
their instruction to meet students’ needs (Gewertz, 2010; Stiggins, 2010). Assessment also 
provides feedback to students in order for them to shape their understanding and improve their 
learning. Formative assessment helps students answer the question, “How am I doing?” as they 
are learning new material, while summative assessment answers the question, “How did I do?” at 
the end of learning a unit (Shute, 2008; Stiggins, 2005). 

Recent meta-analyses have documented the effectiveness on the use of assessment 
practices on student achievement (Black & Wiliam, 1998; Hattie & Temberley, 2007; Kluger & 


3 



Journal for Learning through the Arts, 10(1) (2014) 


DeNisi, 1996). Black and Wiliam (1998) examined 250 studies from the research addressing a 
range of student factors and teacher instructional practices, including formative assessment 
strategies. They concluded that fonnative assessment has a more profound effect on learning 
than do other typical educational interventions, finding effect sizes between 0.4 and 0.7. 
Moreover, they concluded that assessment practices have a stronger effect on low achieving 
students than on high achieving ones, as they are instrumental in developing meta-cognitive 
skills and enhancing motivation. Hattie and Temberley’s (2007) research described the results of 
12 meta-analyses that included 196 studies and 6,972 effect sizes. Notably, they found that the 
average effect size for the use of formative feedback was 0.79. 

Arts Achieve Project Implementation 

The Arts Achieve project includes a partnership among the NYCDOE OASP, five of the 
city’s premier arts organizations, and Metis Associates, the project’s evaluator. The arts 
organizations and their particular arts focus for the project are as follows: STUDIO (lead partner, 
visual arts); ArtsConnection (theater); the Dance Education Laboratory at the 92nd Street Y 
(dance); the Weill Music Institute at Carnegie Hall (music), and the Cooper Hewitt National 
Design Museum (technology). Figure 2 provides a description of each of the partner 
organizations. Prior to the first year of project implementation, the NYCDOE and the partner 
organizations developed and piloted 12 Benchmark Arts Assessments, one in each arts discipline 
(dance, music, theater, and visual arts) and school level (elementary, middle, and high). The 
Benchmark Arts Assessments measure students’ arts content knowledge and perfonnance skills 
based on local arts standards set in the NYCDOE Blueprints for Teaching and Learning in the 
Arts. 1 The development of the Benchmark Arts Assessments is described further in the Methods 
section. 

The Arts Achieve project provides professional development to arts teachers over a three- 
year period to help them engage in action research and to learn to use the data from the 
Benchmark Arts Assessments and ongoing formative assessments. Starting in the 2011-2012 
school year, the arts teachers fonned art discipline-based PLCs to work together toward the goal 
of improving their practice and student learning. Teaching artists from the arts organizations 
were partnered with the participating arts teachers to work together over the course of the 
project. In contrast to an artist-in-residence program, the teaching artist’s role is that of a 
facilitator, who helps support the arts teacher with project work, as well as plan for instruction 
and provide assistance in the classroom. The Arts Achieve in-service professional development 
includes: 

• On-site Consultancies'. Each facilitator visits his/her partner school about twice a 
month over the course of the school year, for a total of 22 visits over the course of 
the school year. During each of these visits, the facilitator and arts teacher work 
directly with two of the teacher’s targeted art classes 2 and participate in a separate 
joint instructional planning meeting. While the specific activities during each visit 
are unique to the needs of the arts teacher, the facilitator’s time in the classroom 
may include observing, modeling, or co-teaching. During the planning time, the arts 
teacher and facilitator may discuss the teacher’s action research and frequently 
review student work, examine fonnative and summative assessment data, and 
discuss the instructional implications of data. 

• Assessment retreats'. The facilitators and arts teachers meet three times over the 
course of the school year as a full group to focus on the appropriate use of 
fonnative and summative assessment strategies and the use of data from the 
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Benchmark Arts Assessments to inform instruction. The assessment retreats are 
facilitated by Dr. Heidi Andrade, an expert in fonnative assessment practices. These 
meetings also provide an opportunity for participants to meet within their discipline 
and grade level specific PLCs. 

• Inter-visitations : Two school inter-visitations per year provide additional 
opportunities for participants to meet within their PLCs. During these inter¬ 
visitations, the arts teachers deepen the work of the PLCs by discussing student 
work and sharing best practices around instruction and use of formative assessment 
strategies. The initial visit was to a model school, while all subsequent visits have 
been to other treatment schools. 

• Online Community. To support ongoing collaboration and share ideas and 
resources, a social-networking site was created for the project using Ning. This 
online community allows the Arts Achieve participants to share student work by 
posting pictures or videos, to upload units of study or links to websites, to network 
with other teachers in the project, to make announcements, and to archive project 
documents. 

In addition, the Arts Achieve project provides participating arts teachers with resources to 
support this work. Resources include Blueprint- aligned exemplary units of study, links to 
websites and other sources, their students’ results from the Benchmark Arts Assessments, and 
technology bundles. The technology bundles include three iPads and a projector to use in their 
classrooms. The arts teachers participated in a full-day of training on how to operate and 
incorporate the iPads into their instruction. Through the incorporation of these additional 
resources, teachers will have increased access to arts content, formative and summative 
assessment strategies, and instructional strategies designed to reach students through multiple 
venues. 

The Arts Achieve project also provides the participating facilitators with additional 
training each year of the project. In the first year, three days of training were held—two prior to 
the start of implementation in the schools and one during the school year. In subsequent years, 
additional training days have been provided. During these trainings, the facilitators are provided 
with techniques and strategies on how to work with their partner arts teachers, apply formative 
assessment in the classroom, and use student data from fonnative assessments and the 
Benchmark Arts Assessments to improve instructional practice. 

Purpose of Study 

The literature on the positive impact of formative assessment, action research, and PLCs 
on teacher instruction and student achievement, as described above, provides ample evidence to 
support the Arts Achieve framework. However, the literature is limited in that it does not provide 
evidence on specific impact on arts teacher instruction or student achievement in the arts. The 
purpose of this study is to extend and deepen this previous research, and to examine whether, and 
to what extent, providing teachers with high quality and intensive professional development can 
positively impact arts teachers and their students. 

Overall, the Arts Achieve study addresses the following three research questions: 

1. What is the nature of Arts Achieve implementation? What are the successes and 
challenges of Arts Achieve implementation? 

2. What is the impact of the Arts Achieve project on arts teachers’ Blueprint knowledge and 
instructional practices, including their use of the Blueprint standards and their 
interpretation and use of formative and summative assessment data? 


5 



Journal for Learning through the Arts, 10(1) (2014) 


3. What is the impact of the Arts Achieve project on students’ arts achievement? Are there 

differential impacts of the Arts Achieve project by arts discipline and school level? 

Methods 

To evaluate the impact of the Arts Achieve project, Metis Associates designed a cluster 
randomized control trial, whereby schools were assigned to either the treatment or status-quo 
control condition. This design allows for comparisons between teachers and students who 
participate in the treatment with those who do not. This section describes how the sample was 
selected, the instruments used to collect teacher and student background characteristics and 
outcomes, and the analyses used to address the study’s research questions. 

Sample 

The Arts Achieve sample was selected using a two-stage process. First, NYCDOE public 
schools that met the basic eligibility requirements were recruited and then randomly selected to 
participate. To identify eligible schools, infonnation about their arts programs was gathered 
through the 2009-2010 NYCDOE Annual Arts Education Survey . 3 To be eligible to participate, 
schools had to be part of the NYCDOE public school system and have an existing arts program 
in at least one arts discipline. Additionally, eligible elementary schools had to: 1) serve all grades 
3-5; 2) offer at least 30 hours of instruction in one of the arts disciplines to all grades 3-5; and 3) 
have a certified or a cluster teacher assigned to teach the arts. Eligible middle schools had to: 1) 
serve both grades 7 and 8; 2) offer at least one full year of instruction in an arts discipline to 
eighth-grade students; and 3) have an arts teacher on staff certified to teach the arts. Eligible high 
schools had to: 1) serve any grade 9-12; 2) offer at least one full year of instruction in an arts 
discipline to students in any grade 9-12; and 3) have an arts teacher on staff certified to teach the 
arts. 

Two informational meetings in winter 2011 were held to describe the program and to 
recruit volunteer schools to participate in the study. Schools recruited for participation were 
blocked on two characteristics—arts discipline (dance, music, theater, and visual arts) and school 
level (elementary, middle, and high schools)—thereby creating 12 blocks. In cases where there 
were more eligible schools than needed, schools were randomly selected to participate. 

In the second stage of the selection process, schools were randomly assigned to the 
treatment or control group. Within each arts discipline block, eight schools were selected at the 
elementary level, six at the middle school level, and six at the high school level. At the 
elementary school level, five schools were assigned to the treatment group and three schools 
were assigned to the control group. At the middle and high school level, three schools were 
assigned to the treatment group and three schools were assigned to the control group. 

Table 1 displays the number of participating schools, arts teachers, and students who 
participated in the first implementation year. The data are categorized by arts discipline, school 
level, and group. A total of 77 schools, 43 treatment and 34 control, participated in the first year 
of implementation. Because of shortages in the number of eligible dance and theater programs, 
three schools were in more than one block. One school with both elementary and middle school 
grades offered instruction in dance and was randomly assigned to the control group at both 
levels. A middle school that offered instruction in both theater and visual arts was randomly 
assigned to the theater treatment group and then randomly assigned to the visual arts control 
group. In this case, the school was pulled out as a visual arts control school. One high school 
with both dance and theater programs was randomly assigned to both the dance and theater 
treatment groups. Lastly, one theater treatment elementary school and one theater control high 
school elected not to participate prior to the start of implementation. 
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The treatment was targeted mainly toward the school’s arts teacher in the arts discipline 
upon which the school was blocked. Within the treatment schools, the arts teachers’ targeted 
classes in the treatment and control schools were selected based on grade (grade 5 in elementary, 
grade 8 in middle school, 4 and grades 9-12 in high school); year-long instruction (all target 
classes had to participate in year-long arts instruction); and (if multiple classes met these eligible 
requirements) scheduling convenience. The classes selected for assessment in the control schools 
were selected using the same criteria. Arts teachers in the treatment schools participated in all the 
Arts Achieve professional development, whereas the arts teachers in the control schools 
participated only in evaluation activities. 

In general, one arts teacher per school participated in the treatment, although there are 
some schools with more than one participating arts teacher. A total of 79 arts teachers 
participated in the project, 44 arts teachers from the treatment schools and 35 arts teachers from 
the control schools. A total of 4,066 students received yearlong instruction in art within the arts 
teachers’ targeted classes, including 2,046 students in the treatment schools and 2,020 students in 
the control schools. Table 2 displays the demographic profile of participating students by group. 

Twenty-four teaching artists, who all had previous experience working in schools, 
participated in the project as facilitators (six dance, five music, seven theater, and six visual arts). 
The majority of facilitators were partnered with only one arts teacher; however, some worked 
with two or three teachers. 

Instruments 

Ongoing data collection for Arts Achieve consists of program documentation, surveys and 
focus groups with arts teachers, and the Benchmark Arts Assessments. Additionally, secondary 
data are collected from the NYCDOE, including students’ background characteristics, and 
English Language Arts achievement. This section describes the Arts Achieve data collection 
procedures and instruments. 

Program documentation. To measure Arts Achieve implementation, program 
documentation is collected, including professional development materials, participant 
attendance, and observations of the trainings. Data on the number of on-site consultancies also 
are collected. 

Arts teacher surveys. To collect information on arts teachers’ knowledge and 
instructional skills, Metis and program staff developed surveys and administered them to the 
participating arts teachers in the treatment and control schools at the beginning (pre) and end 
(post) of the school year. The surveys ask questions about arts teachers’ characteristics, including 
their years of teaching experience and certification in the designated arts discipline. The surveys 
also use Likert-scale questions to ask about arts teachers’ knowledge and use of the NYCDOE 
Blueprints and use of formative and summative assessment data. Open-ended questions were 
also included to learn about the successes and challenges of implementing the Arts Achieve work 
in the school. Thirty-six out of forty-four (81.8%) of the treatment arts teachers responded to 
both the pre- and post-surveys, while 17 out of 35 (48.6%) of the control arts teachers responded 
to both surveys. 

Composite scores of teachers’ Blueprint knowledge, Blueprint use, and formative 
assessment strategies were calculated using multiple items on the arts teacher pre- and post¬ 
surveys. The Blueprint knowledge scale consisted of five items and had a Cronbach’s alpha 
value of 0.77 on the post-survey. The Blueprint use and use of formative assessment strategies 
post-survey scales were also found to have high internal consistencies (4 items a=0.79, 3 items 
a=0.73, respectively). 
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Benchmark Arts Assessments. As described above, the Benchmark Arts Assessments 
were developed in the planning year and were used to measure the impact of Arts Achieve on 
students’ arts achievement. Prior to the first year of implementation, teams were assembled to 
spearhead the creation of the assessments. The teams were led by the NYCDOE Arts Directors in 
the four arts forms, and also were comprised of school-based arts teachers, staff from the 
participating arts organizations, and NYCDOE experts in test and measurement. A total of 12 
assessments were developed (three [one each for 5 th grade, middle school, and high school] in 
each of the four arts disciplines [dance, music, theater, and visual arts]). The assessments were 
designed to measure the extent to which students have developed the knowledge and skills that 
are expected in a particular art form by fifth grade, middle school, and high school, according to 
the NYCDOE Blueprints. They also have been aligned with the Common Core Capacities in 
English Language Arts. 

The Benchmark Arts Assessments each have multiple components, including 
performance and written sections, and include activities that address content knowledge, 
transferable concepts, and skills in the designated arts discipline. The assessments allow students 
to demonstrate their depth of knowledge in the content area through the analysis of 
performances, evaluation of other masterworks, and the creation of their own works of art. 
Question types include short-answer, fill-in-the-blank, multiple choice, and perfonnance tasks. 
The majority of the questions are scored using a four-point scaled rubric. 

Prior to each administration of the assessments, Arts Achieve staff provided training for 
adjudicators in administering and scoring the assessments. The adjudicators were current or 
retired NYCDOE arts teachers or facilitators. Separate training was conducted for each of the 
arts disciplines, but each included background on the context and purpose of the Benchmark Arts 
Assessments and training on the rubrics to score student work. Two adjudicators each 
administered and scored the assessments. 

In spring 2011, the assessments were piloted in a sample of NYCDOE public schools that 
met the same requirements as the study schools, and the psychometric properties of the 
assessments were analyzed. Based on the results of the pilot, the assessments were found to have 
acceptable levels of reliability and validity. The assessments are administered to students in 
participating classes in the treatment and control schools in both the beginning (pre) and end 
(post) of the school year. Total scores for each of the assessments use a scale of 0 to 100. 

Table 3 displays the internal consistencies of each of the spring 2012 Benchmark Arts 
Assessments. The Cronbach’s alpha values range from 0.72 on the elementary school music- 
vocal assessment to 0.89 on the high school theater-direction assessment, indicating that the 
internal consistencies are acceptable. Inter-rater reliabilities were calculated for each task scored 
using a rubric on each of the assessments. The Kappa values ranged from 0.13 on the high school 
dance assessment, group dance perfonnance collaboration task to 1.00 on the music elementary 
assessment, vocal and instrumental performance tasks. The appendix presents the reliabilities for 
each task. 

Secondary data. Data on treatment and control students’ characteristics were obtained from 
the NYCDOE to use as covariates in the analysis models. Student characteristics include student 
demographics (including gender, race/ethnicity, English language learner status, special 
education status, free/reduced priced lunch status), and average daily attendance. Students’ 
scores on the New York State English Language Arts (NYS ELA) exam also were collected as a 
measure of student academic achievement. In elementary school and middle school, the NYS 
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ELA assessment is administered to all students in grades three through eight in the spring of each 
school year. 

Data Analysis 

Descriptive statistics were calculated, including frequencies, means, and standard 
deviations, on the attendance, survey, and arts achievement data. Multiple regression analyses 
were conducted to measure the impact of the Arts Achieve project on arts teachers’ knowledge 
and instructional practice scores, as measured by the arts teacher post-survey and students’ arts 
achievement scores on the post-Benchmark Arts Assessments. Potential confounding factors or 
covariates that have a relationship with the treatment or outcome are included in each regression 
model to reduce threats to the study’s internal validity. Covariates included in the teacher 
outcome models included arts teachers’ years of experience, arts discipline certification, and pre¬ 
survey composite scores. In the student arts achievement models, student demographics (i.e., 
gender, race, English language learner status, special education status, free and reduced price 
lunch status, and average daily attendance) and prior achievement (i.e., spring 2011 NYS ELA 
exam score and pre-Benchmark Arts Assessment score) were used as covariates. All covariates 
included in the multiple regression models were grand mean centered. For statistically significant 
impact effects, Glass’s delta was calculated to obtain the impact’s effect size. 

Results 

As stated above, the Arts Achieve project is based on the theory that participation in the 
Arts Achieve professional development will have positive impacts on arts teachers’ knowledge 
and instructional skills and, in turn, on students’ arts achievement. This section presents 
preliminary implementation and impact findings of the project after one full year of 
implementation. 

Implementation Findings 

As described earlier, the participating arts teachers and their partner teaching artists, or 
facilitators, formed PLCs to work together to engage in action research and to leam to use data 
from formative assessments and the Benchmark Arts Assessments. The facilitators collaborated 
with their partner arts teacher to identify gaps in student learning and devise an action plan that 
would address these gaps. The facilitator played the role of “critical friend” to the arts teachers, 
and did not necessarily work with the students directly (though this may have been an element of 
the work). During the first year of implementation, facilitators were expected to conduct a total 
of 22 on-site consultancies over the course of the school year. Each consultancy included an 
extended planning session with the teacher to discuss strategies, examine data, debrief on 
previous meetings, and review student work. 

Table 4 displays the average number of Arts Achieve on-site consultancies conducted by 
facilitators in Year 1. The data are presented by arts discipline and school level. Across all 
treatment schools, the facilitators and arts teachers conducted an average of 19.6 (SD=2.25) 
consultancies, which is slightly less than the expected 22. The average number of consultancies 
ranged from 18.0 (SD=1.53) in the music middle schools to 21.3 (SD=1.00) in the dance and 
visual arts high schools. Facilitators explained that scheduling the consultancies with their 
partner arts teachers was often a challenge due to other commitments on both of their parts. 
Facilitators often had restrictions on the days that they could visit due to residency commitments 
in other schools, and arts teachers often had other commitments during the school day as well. 
Additionally, the art schedule in the buildings often was disrupted due to testing, school trips, 
and other activities. 
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Despite the challenges in scheduling, the consultancies, data collected through surveys 
and focus groups indicate that the arts teachers and facilitators were largely successful in 
fostering strong partnerships. Arts teachers were highly positive overall about their facilitators 
and appreciated the opportunity to collaborate with other artists. For example, on an end-of-year 
survey, one teacher wrote, “My partner facilitator was very helpful, encouraging, reflective, 
intelligent, and I couldn’t imagine how she could have been more helpful. She was a great 
partner... [a] first rate artist, instructor, [and] mentor.” Another teacher explained that her 
facilitator was an excellent resource for planning instruction. The consultancy planning time 
allowed her opportunities to bounce ideas off a fellow artist educator. These types of 
opportunities, in general, were perceived as critical, given that the great majority of arts teachers 
have no school-based peers with whom to collaborate. The only critique that many of the 
participating arts teachers expressed was that they wished they had more time to meet and plan 
with their facilitators. 

In addition to the consultancies, the Arts Achieve project allows for other opportunities 
for the facilitators and their partner teachers to collaborate. In Year 1, these opportunities 
included three full-day assessment retreats and two school inter-visitations. Additionally, there 
were three training days designed for facilitators only. Table 5 displays participants’ attendance 
at these professional development activities in Year 1. The majority of arts teachers and 
facilitators participated in all four assessment retreats over the course of the year; however, the 
great majority of arts teachers and facilitators did not attend both of the inter-visitations 
scheduled during the first year. While most teachers attended the first inter-visitation in fall 2011 
to observe best practices in a model school, a smaller number with scheduling conflicts were able 
to attend the spring inter-visitation at one of their peer’s schools. Despite scheduling challenges, 
both facilitators and arts teachers reported that these professional development opportunities 
were extremely useful in developing their skills and knowledge in formative assessment 
practices. Arts teachers valued the time to meet in their PLCs and to continue the planning that 
they had worked on during their consultancies. Likewise, facilitators provided positive feedback 
on the professional development sessions targeted for them specifically. Initially, many felt 
challenged by their new role as facilitator in participating schools and struggled to develop 
trusting and productive relationships with their partner teachers. These sessions allowed them to 
form their own PLCs and to share successes and challenges in their work with teachers, as well 
as to learn the expectations and best practices associated with their role as facilitator. 

Impact Findings 

Arts teacher knowledge and instructional practice. Changes in arts teacher knowledge 
and instructional practice were measured through closed- and open-ended items on pre and post 
self-report surveys. Figure 3 displays data from the closed-ended items, including treatment and 
control arts teachers’ perceptions about their content knowledge and instructional practices at 
pre- and post-administration. The data show that after one year of implementation, arts teachers 
in both the treatment and control schools made some growth in their instructional knowledge and 
skills over the course of the school year. From pre- to post-survey administration, arts teachers in 
the treatment schools had slight increases in their perceptions of their Blueprint knowledge (M pre 
= 3.48, SDp r e = 0.41; M post = 3.55, SD post = 0.35) and Blueprint use in instruction (M pre = 3.37, 
SD pre = 0.96; M P o S t = 4.15, SD post = 0.81). Similarly, arts teachers in the treatment schools 
reported more use of fonnative assessment strategies (M pre = 3.99, SD pre = 1.05; M post = 4.95, 
SD pos t = 0.84) and a better understanding of the importance of reviewing and analyzing data for 
instruction (M pre = 6.79, SD pre = 2.99; M pos t= 7.72, SD pos t = 2.30) over the course of the school 
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year. However, when compared to gains made by the arts teachers in the control schools, the 
results of multiple regression analyses indicated that there was no statistically significant (p > 

.05) effect of treatment on arts teachers’ knowledge or instructional practices, after controlling 
for teachers’ years of experience and arts certification. 

While the regression analyses did not find statistically significant differences between the 
treatment and control groups, data from the open-ended portions of the treatment teachers’ post¬ 
surveys provide evidence to support the impact of Arts Achieve on teachers’ knowledge and 
skills. One teacher explained that the project was allowing her to perfect her craft as a teacher. 
Another teacher explained that he was more focused. This teacher elaborated that through his 
review of the data, he knew where his students’ weaknesses were and that they were aligned with 
the instructional areas on which he had not placed focus over the course of the year. He further 
explained that he planned to re-focus his instruction in the coming year. Another teacher 
summarized her learning by saying, “I am very grateful to be involved with Arts Achieve. I have 
learned a lot about my teaching, and am so excited that there are so many ways that I am 
improving my craft,” and elaborated that she looked forward to continuing her growth over the 
years of the project. 

Student arts achievement. Multiple regression analyses were conducted to measure the 
impact of the Arts Achieve project on students’ arts achievement after controlling for student 
demographics (i.e., gender, race, English language learner status, special education status, free 
and reduced priced lunch status, and average daily attendance) and previous achievement (i.e., 
spring 2011 NYS ELA exam score and pre-Benchmark Arts Assessment score). Table 6 displays 
the results of the multiple regression analysis on students’ arts achievement across all arts 
disciplines and school levels. The results indicate that, after controlling for differences in 
demographics and previous achievement, the participating students in the treatment schools 
performed 3.11 points higher on the post-Benchmark Arts Assessments than participating 
students in the control schools after one year of implementation. The predicted mean post- 
Benchmark Arts Assessment score for the average treatment students is 62.7 as compared to 59.6 
for the average control student. The Glass’s delta indicated that there was a small effect size of 
0.18. 

To detennine if there were differential impacts by arts discipline and school level, 
separate regression models were conducted after selecting the appropriate subsample. Table 7 
displays the results of the multiple regression analyses on students’ arts achievement by arts 
discipline. After controlling for demographic and previous achievement differences, the results 
indicate that, across all grade levels, participating students in the treatment schools scored higher 
on the post-Benchmark Arts Assessments than participating students in the control schools in 
music, theater, and visual arts. Across all school levels, the theater schools had the largest 
statistically significant (p < .05) treatment effect with an effect size of 0.42. Participating 
students in the theater treatment schools perfonned 7.44 points higher on the post-Benchmark 
Arts Assessments than participating students in the control schools. The predicted mean post- 
Benchmark Arts Assessment score for the average treatment theater students is 60.3 as compared 
to 52.8 for the average control student. The music and visual arts treatment effects also were 
statistically significant (p < .05) with effect sizes of 0.26 and 0.21, respectively. The average 
treatment music student had a predicted mean post-Benchmark Arts Assessment score of 68.5, 
whereas the average control music student mean is 64.6. The average treatment visual arts 
student had a predicted mean post-Benchmark Arts Assessment score of 56.7, whereas the 
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average control music student mean is 53.85. The dance treatment effect is approaching 
significance, p = .05, with a Glass’s delta of 0.11. 

Table 8 displays the results of the multiple regression analyses on students’ arts 
achievement by school level. Across arts disciplines, the results of the multiple regression 
analyses indicate that participating students in the treatment elementary and high schools scored 
higher on the post-Benchmark Arts Assessments as compared to students in the control 
elementary and high schools. Participating students in the treatment elementary schools 
performed 3.79 points higher on the post-Benchmark Arts Assessments than participating 
students in the control elementary schools. Participating students in the treatment high schools 
performed 6.66 points higher on the post-Benchmark Arts Assessments than participating 
students in the control high schools. The elementary and high school treatment effects were 
statistically significant (p < .05) with effect sizes of 0.25 and 0.34, respectively. The treatment 
effect for participating students in the middle schools was not statistically significant (p > .05). 

Discussion 

The Arts Achieve project was designed with the goal of improving the quality of arts 
teachers’ instruction through intensive and targeted professional development, which, in turn, is 
intended to lead to an increase in student arts achievement. After the first of three years of 
implementation, preliminary results provide promising indication of the impact of the project on 
student arts achievement. 

Arts Achieve was developed based on literature that connects improved student 
achievement with increased teacher use of formative and summative assessment and data-driven 
decision making practices. While evidence on the effectiveness of these practices is abundant 
across education, little (if any) research on its impact on arts teachers exists. The reason for this 
lies in the fact that there are few assessments in the arts and, as such, arts teachers tend not to use 
data to guide their instruction. Furthennore, arts teachers generally have not been trained in the 
use of data-driven decision making. 

The Arts Achieve project began with the intense, but ultimately rewarding, process of 
developing arts assessments. During the first project year (which was used for planning and 
development), 12 Benchmark Arts Assessments were created, one for each of the four arts 
disciplines and three school levels. The Benchmark Arts Assessments are aligned with the NYC 
DOE’s Blueprints and represent an important step in defining and measuring expectations for 
teaching and learning in the arts. The assessments, which largely have been found to have 
acceptable levels of reliability and validity, mark an important deliverable for the grant. 

However, the ambitious Arts Achieve project aimed to do much more than simply create 
the assessments. The project aimed to provide intensive professional development for teachers, 
which includes the use of data from the Benchmark Arts Assessments, to help them shape their 
instruction, and ultimately improve student arts achievement. 

This article provides data on implementation and impact after one full year of 
implementation according to the three main research questions addressed by the Arts Achieve 
study, including: 1) successes and challenges of implementation; 2) impact of the project on 
teachers’ knowledge and skills; and 3) impact of the project on students’ arts achievement. 
Successes and Challenges 

As described above, a major success of the project was the creation of 12 reliable and 
valid Benchmark Arts Assessments. Additionally, strong partnerships between arts teachers and 
facilitators were forged, fostering growth for both groups. The project required both groups to 
move outside of their comfort zone—facilitators had to play a role in schools that was often 
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brand new to them, while teachers had to learn to dig into data and to reflect on their 
instructional implications. The pairings had to develop strong, trusting relationships that would 
ultimately lead to productive change in student learning. 

Challenges confronted were not atypical to programs implemented in urban school 
environments. Teachers in general—and arts teachers are no exception—have extreme demands 
on their time. There are constant distractions within the schools, and oftentimes, art can take a 
backseat to content areas that are subject to high-stakes testing. As a result, fewer than expected 
on-site consultancies took place over the course of the year, and a small percentage of arts 
teachers and facilitators were able to participate in both of the planned school inter-visitations. 
Furthermore, while most of the partnerships were strong, a few suffered from trust issues, 
teacher reluctance to change their practices, and/or personality conflicts. 

Based on the findings from the first year of implementation, the project has made some 
adjustments to the design, including fewer in-school visits and more assessment retreat time. 
Additionally, the role of the facilitator has been better communicated to participants, and 
teachers and facilitators were given more time in the second year of implementation to 
strengthen their relationships, as well as to work together on action research. 

Impact on Arts Teachers’ Knowledge and Skills 

Impact of the project on arts teachers’ knowledge and skills was measured through self- 
report surveys, which were distributed to teachers in both the treatment and control groups on a 
pre and post basis. While teachers indicated in open-ended portions of the survey that the project 
was highly impactful to them, results of multiple regression that examined differences in growth 
between treatment and control teachers did not detect statistically significant differences. Given 
the contrast in findings, it is possible that the scales designed to measure teacher knowledge and 
skills are not sensitive enough to detect change over the course of the school year. It is also 
possible that additional implementation time is needed to detect an impact in teacher growth. 
Impact on Student Arts Achievement 

Findings from the analyses of student arts achievement indicated that, across all grade 
levels and arts disciplines, students who received yearlong arts instruction in the treatment 
schools demonstrated greater growth in their arts achievement than students who received 
yearlong arts instruction in the control schools. This finding supports the hypothesis that the 
work with teachers ultimately leads to improvement in student growth. 

Interestingly, however, given that our theory of change is that professional development 
will positively impact teacher knowledge and skills that, in turn, will lead to increases in student 
arts achievement, the fact that there are significant positive effects for students’ arts achievement 
but not for teacher growth, further suggests that the measure through which teacher growth is 
assessed is not sensitive enough to detect change. In coming years, teacher growth will be 
explored in greater depth, and additional tools, including qualitative methods, will be used to 
gather data. 

It is also notable that there were differential impacts on student achievement according to 
arts discipline and school level. These results will be explored in greater detail in coming years 
to examine whether patterns persist and, if so, what factors are contributing to the differential 
impact. 

Overall, findings from the first year of implementation of Arts Achieve provide highly 
promising results. Though it is still in progress, the project has already produced assessments that 
have the potential to be valuable to the field. Furthermore, the study provides strong evidence 
that increasing arts teachers’ knowledge and skills in formative and summative assessment and 
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the use of data leads to improvement in student arts achievement. It has furthermore raised new 
questions and areas of inquiry into which the field may explore in the future. 
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Notes 

1 The Blueprints set clear standards for what students should know, understand, and be able to do 
in each of the four arts disciplines (dance, music, theater, and visual arts) as they move through 
the school system from Pre-K through 12 th grade. They are based on National arts standards and 
support the NYS Standards for Arts Instruction. Scope and sequence of learning are identified on 
the Blueprints through five strands: Art Making, Literacy in the Arts, Making Connections, 
Community and Cultural Resources, and Careers and Lifelong Learning. Benchmarks are 
delineated at four levels - Grades two, five, eight, and 12. 

2 In elementary schools, fifth-grade classes are targeted. In middle schools, eighth-grade classes 
are targeted, though some sixth- and seventh-grade classes may be included. In high school, the 
targeted classes can include any grade from nine through 12. 

3 Each spring all, NYCDOE schools are asked to complete the Annual Arts Education Survey. 
The survey collects information on schools’ arts programs. More specifically, the survey 
includes questions about student participation in arts courses by arts discipline, arts activities 
conducted, and certified school-based arts teachers. 

4 In some cases, if eighth grade could not be targeted, the treatment was geared toward grade six 
or seven. 
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Table 1 


Number of Arts Achieve Schools, Teachers, and Students by Arts Discipline, School Level, and 
Group _ 


Art 

Discipline 

School 


Treatment 



Control 


Level 

N 

Schools 

N 

Arts 

Teachers 

N 

Students 

N 

Schools 

N 

Arts 

Teachers 

N 

Students 

Dance 

Elementary 

3 

5 

212 

3 a 

3 

193 


Middle 

3 

3 

117 

3 a 

3 

160 


High 

3 b 

3 

98 

3 

3 

135 

Dance Subtotal 

11 

11 

427 

9 

8 C 

488 

Music 

Elementary 

5 

5 

185 

3 

3 

209 


Middle 

3 

3 

195 

3 

3 

289 


High 

3 

3 

181 

3 

5 

186 

Music Subtotal 

11 

11 

561 

9 

11 

684 

Theater 

Elementary 

4 d 

4 

238 

3 

3 

128 


Middle 

3 

3 

66 e 

3 

3 

150 


High 

3° 

4 

103 

2 f 

2 

92 

Theater Subtotal 

10 

11 

407 

8 

8 

370 

Visual Arts 

Elementary 

5 

5 

340 

3 

3 

225 


Middle 

3 

3 

153 

2 g 

2 

91 


High 

3 

3 

158 

3 

3 

162 

Visual Arts Subtotal 

11 

11 

651 

8 

8 

478 

Overall Total 

43 

44 

2,046 

34 

35 

2,020 


a Although there are three control elementary schools with a dance program and three control 
middle schools with a dance program, one school is included in both groups. Consequently, there 
are a total of eight control schools with a dance program, not nine schools. 

h Although there are three treatment high schools with a theater program and three treatment high 
schools with a dance program, one school is included in both groups. Consequently, in the two 
groups, there are a total of five high schools, not six high schools. 

c Although there are three control elementary school dance teachers and three control middle 
school dance teachers, one teacher is included in both groups. Consequently, there are a total of 
eight control school dance teachers, not nine schools. 

d A theater treatment elementary school elected not to participate in the project prior to the start 
of implementation. 

e A treatment theater middle school did not provide students with yearlong theater instruction, 
therefore, this number represents the number of students in two of the treatment theater middle 
schools. 

f" 

A theater control high school elected not to participate in the project prior to the start of 
implementation. 

g A visual arts control middle school was initially randomly assigned as a treatment theater 
middle school; therefore the school was pulled out of the study as a control school for visual arts. 
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To control for diffusion of treatment effects, study schools cannot participate as both treatment 
and control schools. 
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Table 2 


Demographic Profile of Participating Students by Group 


Student Characteristic 

Treatment 

Control 

N Students 

2,046 

2,020 

Female 

54.8% 

56.5% 

Race 



White 

14.9% 

14.6% 

African American 

28.6% 

31.5% 

Asian 

18.5% 

17.7% 

Hispanic 

37.0% 

35.7% 

Other 

0.93% 

0.45% 

English Language Learner 

14.0% 

8.7% 

Special Education 

13.9% 

11.4% 

Free-Reduce Priced Lunch 

83.8% 

79.3% 

Average Daily Attendance (SD) 

93.5 (10.58) 

92.8 (10.21) 

Spring 2011 NYSELA Proficiency 3 

50.4% 

55.6% 


a Students’ spring 2011 NYSELA proficiency level was not available for all participating 
students. There were 1,618 treatment students with spring 2011 NYSELA scores and 1,467 
control students with scores. 
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Table 3 


Internal Consistencies of the Spring 2012 Benchmark Arts Assessments 



Benchmark Arts Assessment 




Art Discipline 

School Level 

Subtest 3 

N Cases 

N Items 

a 

Dance 

Elementary 


286 

16 

0.85 


Middle 


214 

15 

0.88 


High 


120 

19 

0.84 

Music 

Elementary 

Instrumental 

88 

15 

0.76 



Vocal 

203 

15 

0.72 


Middle 


319 

23 

0.86 


High 


225 

26 

0.83 

Theater 

Elementary 

Playwriting 

127 

12 

0.85 



Costume Design 

85 

13 

0.83 


Middle 

Acting - Actors 

65 

9 

0.81 



Musical Theater - Actors 

25 

7 

0.77 



Playwriting - Actors 

18 

7 

0.78 



Acting - Design 

1 

7 

- 



Musical Theater - Design 

5 

7 

- 



Playwriting - Design 

0 

7 

- 


High 

Direction 

132 

10 

0.89 



Acting 

39 

11 

0.77 

Visual Arts 

Elementary 


441 

29 

0.85 


Middle 


228 

25 

0.88 


High 


183 

18 

0.87 


3 The music elementary assessment and each of the theater assessments asked students to choose 
between tasks on the assessments. In these cases, the internal consistencies are presented for each 
of the subtests. 
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Table 4 


Average Number of Arts Achieve On-site Consultancies by Arts Discipline and School Level 


Art Discipline 

School Level 

N Schools 

Mean 

SD 

Dance 

Elementary 

3 

19.4 

3.91 


Middle 

3 

20.7 

0.58 


High 

3 

21.3 

1.53 

Dance Subtotal 


11 

20.3 

2.72 

Music 

Elementary 

5 

20.2 

1.48 


Middle 

3 

18.0 

1.00 


High 

3 

20.0 

1.73 

Music Subtotal 


11 

19.5 

1.63 

Theater 

Elementary 

4 

19.5 

1.00 


Middle 

3 

15.7 

2.08 


High 

3 

19.7 

3.06 

Theater Subtotal 


10 

18.4 

2.63 

Visual Arts 

Elementary 

5 

19.6 

1.52 


Middle 

3 

20.0 

1.73 


High 

3 

21.3 

1.53 

Visual Arts Subtotal 


11 

20.2 

1.60 

Overall Total 

43 

19.6 

2.25 
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Table 5 


Arts Teacher and Facilitator Attendance at Professional Development Training 


Professional Development Trainings 

Arts Teachers 

Facilitators 

Assessment Retreats 

Attended All Four Days of Training 

56.8% 

76.7% 

Attended Three Days of Training 

22.7% 

7.0% 

Attended Two Days of Training 

18.2% 

9.3% 

Attended One Day of Training 

2.3% 

4.7% 

Inter-visitations 

Attended All Two Visits 

13.6% 

18.6% 

Attended One Visit 

54.5% 

58.1% 

Facilitator Training 

Attended All Three Days of Training 

- 

58.1% 

Attended Two Days of Training 

- 

16.3% 

Attended One Day of Training 

- 

18.6% 
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Table 6 


Student Arts Achievement Multiple Regression Results (N=2,726) 


Variable 

B 

B SE 

Constant 

59.61** 

0.39 

Pre Arts Achievement Score 

0.32** 

0.02 

Female 

2.30** 

0.54 

Spring 2011 NYSELA Scale Score 

0.16** 

0.01 

Hispanic 3 

-2.53** 

0.64 

African American 3 

-5.39** 

0.68 

Average Daily Attendance 

0.30** 

0.06 

English Language Learner 

-3.47* 

1.05 

Free-Reduce Priced Lunch 

-1.43* 

0.68 

Special Education 

-1.58 

0.84 

Grade 

0.57** 

0.15 

Group 

3.11** 

0.52 

R 2 


0.33 

F 


122.07** 


a The comparison group included white, Asian, or other racial backgrounds. 
*p < .05, **p < .001. 
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